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Since a concept is a general idea about how things should be, 

m 

I will confine my portion of this symposium to that topic, I hope 
that Joe Froomkin will be helpful in showing the problems he has 
encountered in trying to make things as they should be. Then we will 
move to the state level where Bill Miller will be able to present his 
view of things as they are in contrast to how they should be and 
finally, Jim Maxey will present some of the problems he encountered 
while attempting our first go-around at a statistical analysis of 
Title I data. 

This portion will center on information requirements as defined 
by the several levels of educational decision making, the process of 
information gathering, information structuring, information display 
alternatives and information sources in an attempt to present a workable 
evaluation conceptual paradigm. The changes in information needs at 
the various levels of educational decision making and the subsequent 
changes in information requirements have caused much disappointment 
and dissatisfaction with attempts to supply information since the 
reapplication of past evaluation strategies has proved inadequate. Until 
recently evaluation strategies have been the stepchild of educational 
research. Examples of the superficial treatment of die field abound 
in the literature and need not be elaborated here. But, a brief review 



of our changing evaluation needs will help set the context of the 
papers to be presented. 

At the federal level, the current emphasis on program planning 
based upon program objectives has altered significantly the information 
needs of the U. S. Office of Education. This same stress on planning 
at the state educational agency level has been a source of friction 
between those who have the new needs for information and those who 
must provide it. Local educational agencies have also felt the impact 
of federal and state pressure for information and have reacted with 
varying degrees of compliance. Thus, as the decision maker’s role 
in the educational structure has moved from one of stewardship of 
funds and programs to one of change agent through active program 
involvement, the educational information needs of the decision maker 
have also changed. 

The Context 

In order to relate our currentevaluation problems to the context 
of the decision making process, we must first structure the decision 
making process. From structure, the unique information requirements 
at each level can be more clearly understood while the role of the 
common elements of information across each level of the process will 
become more apparent. Decision making is here structured into four 
levels. Each level is followed by a brief description of the context and 



content of the level. 



I ' Long range plannin g: At this level the goals and 
objectives of education are clearly stated, refined 
and clarified. Educational programs are examined 
and elaborated in relation to goals. Also, the 
necessary resources in terms of program requirements 
are estimated. This is the where we intend to go 
level of decision making. 

II. System structuring ; At this level the decision making 
function centers on the structuring of and responsibility 
for transferring goals into working programs and the 
subsequent allotment of resources. This is .he how we 
intend to get there level. 

III. System allocation: At this level decisions concerning 
the elements of staff, facility, finance, curriculum, 
pupil, and community resource allocation and se jaence 
are made. This is what it will cost level. 

IV. System monitoring; At this level decisions based upon 
information provided by the interactions of program 
elements are systematically gathered and presented to 
the relevant decision makers. This is the feedback or 
how we are doing level. 

The four levels of decision making described above have two 
major areas of impact. Internally, decision makers are concerned 
with maintaining a high level of participation, dedication and cooperation 
while minimizing conflict and anxiety. Externally, decision makers 
weigh the effects of the organizational functioning on the many publics 
from the standpoint of gaining support and acceptance. Clearly, these 
conditions call for a continual flow of information. The current 
restructuring of our educational enterprise has created new needs for 
information. 

The differential information needs at each level become even 
clearer when one accepts the fact that educational decisions exist 
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in a federal, state and locaJ context, and have multiple external 
audiences which may, or may not, provide legislative and community 
support for the educational system. 

The present problem of the evaluator is one of setting up a 
methodology that provides relevant information at the appropriate 
time for decisions at multiple levels across varied programs viewed 
by audiences with diverse backgrounds. 

A pause for clarification. It must be understood that information 
systems, whether presented verbally, in printed form, pictorially, 
or in the form of computerized summarizations and manipulations, 
are only instrumentalities designed to convey meaning and intent for 
the facilitation of understanding. Essentially, all information systems 
have sources (the input), ordering, sequencing, classification, and 
analysis procedures (the process) and formats for displaying the 
processed information (the output). The point being that only chance 
can render an information system adequate when the decision maker 
does not specify, before an information system is put into operation, 
exactly what output is expected of the system. To the extent that this 
cannot be done, the system will probably be judged inadequate. 

In the past, educational information typically originated in 
summary form and was further aggregated by district, state, and 
federal officials primarily for control and accounting purposes. 
Generally, the information was adequate for the decisions that had to be 
made, i.e., social bookkeeping, enforcement and regulation of local, 
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state and federal legislation and the distribution of reimbursement 
funds. It was after the fact data for after the fact decisions. Evaluation, 



as conceptualized in the past, differed from our present information 
system concept, centered on the learning process and was seen as 
separate from these information requirements. Our new emphasis 
views education as an integrated system with evaluation in the role of 
monitoring the system. This view is consistent with Weiner’s work 
in cybernetics. 



General Systems Theory 

While evaluators are accustomed to the term system as in 
phrases like ”our educational system, ” their approach to problem 
solving typically follows the classic "statement of the problem" paradigm. 
A brief digression into systems theory will help establish the tone of 
the remainder of this chapter. 

Systems theory will also clarify evaluation’s role by providing 
a frame of reference for program development. Systems concepts and 
lqgic can begin with an oversimplified conceptualization of education as 
the input-black box-output analogy shown in Figure 1 . 



Figure 1 . 
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Input is here defined as the pupils entering the system. 

Pupils can be further described by their sex, age, attitude, aptitude, 
and demographic characteristics as well as by many other factors. 

In systems terminology, these characteristics define the range of 
tolerance for the input "raw material" that enters the system. The 
outputs of the system (graduates, transfers, drop outs) are, in our 
terms, the products of the system. 

The contents "black box" may be represented by five other 
data classification categories. Literally, the black box contains all 
that occurs between the input and the output. Since our concern centers 
on levels of decision making the contents of the box shall be examined 
first in terms of local program monitoring and then in terms of data 
requirements at the other levels. Programs within the Local Educa- 
tional Agency (hereafter called L. E. A. ) can be more clearly 
represented by expanding the box as in a block diagram such as 
Figure 2. There the black box has been expanded to show the existence 
of information overlap or, in analysis of variance terms, confounding 
within and between data categories. 



< o 



7 



Figure 2. 
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Evaluation, in terms of systems methodology, begins with an 



end product or outcome as a given. The delineation of desired output 
is analagous to decision level I on page three. The second step in the 
Hpvelopment of a methodology then consists of a series of logical steps 
to clarify and expand possible alternative means for reaching the output 
and to establish decision rules when a new alternative presents itself. 

The steps are successive and lead back by the expansion of alten v^es 
to input specifications that enhance the success of the alternatives in 
achieving the desired end. l.ie number and variety of the available 
alternatives developed through this process define the complexity of 
the system. Each decision alternative then represents a fork or node 
in the path. The nodes form the logical points for monitoring the system. 

The output at each monitoring point as well as the output at the 
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termination of the complete system is measured against the system 
objectives and performance standards. The quality checks at the 
output of each stage serve to improve the performance of the next 
cycle of the system and are pictured in an oversimplified form as 
Figure 3. 

Figure 3. 
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Evaluation consists of an examination of alternative means 
and their related conditional probabilities at each step of the educational 
process given the existing or desired end . Given a statement of the 
end or outcome, it can be shown that movement (in the sense of defining 
alternate means of achieving an end) is a backward process, a continual 
expansion of possible alternatives and their associated probabilities. 
Clearly, outcome standards serve as the starting points for both the 
building and evaluation of existing as well as proposed programs. The 
paradigm holds for educational systems, missile systems and manu- 
facturing systems. It is as useful as it is generalizable. 



The key concept stated thus far was the probabilistic relation- 
ship of means to ends. In every system, the end or outcome is the 
result of a series of intermediate steps which have the power to 
alter and thus affect the occurrence of the outcome. Therefore, it is 
possible to conceptualize and simulate probability models of existing 
systems and/or proposed systems in many areas. Changes in out- 
come can be predicted as alternatives are selected. This statement 
allows us to consider each refinement of the output statements back 
through the system as a series of conditions, i.e. , necessary and 
sufficient conditions but not causal conditions, which may or may not 
enhance the achievement of the desired output. 

In other words, given a dearly defined end product, a systems 
theorist would proceed back through the decision points and, at each 
step in the procedure, define the limitations (conditional probabilities) 
of the possible alternatives in the described system. The simulation 
possibilities when this approach is applied to evaluating educational 
innovation are in the future, but in the near future. 

The procedure stands in marked contrast to yet enhances the 
usefulness of the hypothesis testing model with its dichotomous accept- 
reject alternatives. Systems theory represents a powerful approach 
to the evaluation problems long associated with the assessment of 
educational programs. Educational evaluation exists today because of 
a commitment to improvement, to enhancement of outcome and to the 
scientific investigation of our attempts to achieve objectives. It is 
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here posited that educational questions such as the assessment of 
a particular program’s impact can be best answered in terms of 
possible contribution to increasing the probability of occurrence of 
defined program outcomes. 

Educational questions are best answered in terms of the 
alternatives that exist for achieving the desired end. Each alternative 
then furnishes the evaluation "raw material" and is measured in terms 
of its contribution to the occurrence of the desired outcome. 

The evaluation strategies presented here attempt to integrate 

information needs and present them as an evaluation product. Since 

education is a change process, the evaluation of education is the 

monitoring of the process through an information system. The thrust 

in evaluation demands a rethinking of our existing information structure 
* 

as a first step in planning the monitoring of change. Educational informa- 
tion can be described by the six interrelated classification areas of 
curriculum, pupil, staff, finance, facility, and community. 

Information from each classification area is interrelated, impunges 
upon the decision making process and is relevant to our problem. The 
author proposes an evaluation methodology that rests upon a core of 



data elements from each of the classification areas gathered primarily 
at the fourth level of decision making. The methodology has four 
features which are basic to meeting information requirements: 1) the 
provision of a base of data elements based upon common definition in 



each data category; 2) a comprehensive picture of the relationships 
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between data content across classification areas; 3) a source of informa- 
tion related to multiple questions across classification areas; 4) a built- 
in set of alternatives as to cost, practice and procedure. 
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Figure 4. 
Curriculum 




It is axiomatic that: 1) one cannot create information from data 
that are not collected; 2) one cannot create information from data 
that are not commonly defined; 3) one cannot anticipate all possible 
information needs. The interrelationships of data elements are shown 
in Figure 4. 

Curriculum 

Curriculum information is placed first as it represents the best 
indicator of the intents of our educational system. For example, one 
aspect of the general goal of a literate population would be a language 
arts strategy at level I, an organizational structure to carry out the 
strategy from decision level II, the further clarification of actual 
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courses such as third grade spelling from level III, and actual course 
outlines and units from level IV. 

The curriculum, then, represents our best estimation of the 
actual content of our educational programs. It is the organizing agent 
in our evaluation system since the grouping of curriculum elements 
from educational programs reflect education’s attempt to achieve 
the goals of our society. The six classifications represent the elements 
necessary for the accomplishment of program objectives. They are 
best thought of as correlates to program effectiveness. Common 
facets of curriculum include: 

1. Course name 

2. Mode of instruction 

3. Method of presentation 

4. Theory of instruction 

5. Time allocation 

Pu pil 

The pupil information classification system consists of data 
elements with common definitions in the areas of demographic character- 
istics, measures the present level of intellectual functioning, and 
measures of achievement, performance, aspiration and expectation are: 

1. Demographic data 

2. Present and past performance in course work 

3. Test information- achievement, intelligence, attitude 

4. Participation in clubs, hobbies, athletics, etc. 

S taff 

The administrative, special education, regular teaching, and 
consultant staff data would be included in this category. Data elements 



13 



are: 

1. Demographic characteristics 

2. Training and experience 

3. Ratings and effectiveness measures 

4. Organizational setting and characteristics 

5. Salary 

Finance 

Financial data related to the maintenance and operation of 
the educational system would be included. Including data elements 
such as: 

1. Tax base and millage 

2. Per pupil expenditures 

3. Administrative overhead 

4. Material and supply expenditures 

Facility 



Common data elements about the facility would include : 

1. Tie site 

2. The buildings 

3. The educational spaces 

4. The equipment 

Community 

The size and composition of the community that each school serves 
would fall in this data category. Typical elements are: 

1 . Area served 

2. Principle industries 

3. Principle housing types and condition 

4. Income level 

5. Ethnic characteristics 

Six classes of interrelated information can then be created from 



the critical data elements from the cate 



cones. 



An evaluation design. 





at least from the standpoint of generalization from a process paradigm, 
would have access to these elements. 

A great deal of work has already been done by the U. S. Office 
of Education on the taxonomy problem for data element definition 
through the handbook series. This effort has centered upon common and 
standard usage of terminology in education; or, if you will, a common 
classification and definition of data elements. The Iowa Educational 
Information Center has also developed many techniques for interrelating 
data elements from the six classification areas. By capitalizing on 
this work, evaluators are now in the position of being able to employ 
data elements that have common definitions across projects, at least 
at tile level of specificity necessary for data aggregation and summariza- 
tion necessary in program comparisons. 

Educational Programs 

The goals of American education are reflected in our educational 
programs, each program represents our operationalization of these 
goals and can be specified under the classification schema. 



Figure 5. 
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The information structure necessary to monitor these programs 
takes the form of a truncated pyramid as shown in Figure 5. The figure 
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is truncated to represent the restriction placed upon data elements 
in each classification category when data is transformed into informa- 
tion at each level in the hierarchy. 

The evaluation process at the level of project monitoring is best 
described as consisting of at least two dimensions. The first is time. 
The time sequence consists of clearly stated (in behavioral terms) 
expected outcomes as written into a proposal, the expected interactions 
leading to the outcomes as defined by those staff members involved 
in the proposal operation, independent observations of the actual 
content and interactions of the program and assessment procedures 
for describing progress toward the achievement of project objectives. 

It is sequential in. that one step follows the other. It is cyclical in that 
the process iterates itself until what is actually happening in the 
program is congruent with the stated implicit and explicit objectives. 
The process can be shown graphically as in Figure 6. 




Figure 6. 
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The process evaluation can be conducted as a series of questions 
which culminate in decision rules. From the written statement of 
objectives, one asks the questions of definition of terms, internal 
consistency, level of specificity and assessment procedures for the 
achievement of stated objectives. These questions can be simply stated. 
"Is this where you want to be at the end of the program?" "How will you 
know that you have arrived there?" 

The second step also consists of a series of inquiries into the 
formulated procedures and interactions intended to achieve the 
objectives and assess progress as they are envisioned by the staff 
of the program. This is usually the statement of behavioral objectives 
at the levels of course outline, specific unite, and the measurement 
devices intended to assess the behaviors. 

The common elements of the program monitoring level as 
evaluative information must meet the criteria of common definition 
of descriptive terms, specified collection times, classification ties, 
and fidelity in relation to each of the six categories specified. 

This is an appropriate point to introduce the problem of informa- 
tion specificity as it relates to level of decision making, i.e., the 
problem of fidelity. Our basic proposition thus far has been that of 
the existence of multiple decision makers. Second, we have asserted 
that decision making can be conceptualized into four levels. Third, 
data elements exist in six related classification areas which can be 
put together in various ways to form information. Fourth, our 



educational system can be divided into program areas that have informa- 
'tion requirements. And, finally, the decision making process consists 
of a series of confrontations between intents, procedures and outcomes 
until a satisfactory state of congruity is reached. All of these 
propositions come together in relation to the question of fidelity. 

Shannon has demonstrated that fidelity (the undistorted inclusion 
of all data) is related to bandwidth (the space available for data 
communication at any one time) in that an increase in bandwidth can 
only be purchased at the cost of fidelity. The converse is also true. 
When these maxims are placed in the evaluation methodology context, 
the data aggregation problem is clarified. 

Complete (high fidelity) information from one project when 
shown in aggregate form with all other project reports represents a 
wide bandwidth report. Individual project fidelity is lost in this 
process. It is the failure to realize that there is an information loss 
inherent in the aggregation of data that has caused problems in state 
and federal reporting procedures. Every time a summarization and 
reordering of data elements are necessary for the next level of the 
decision tree, a loss of individual project fidelity occurs. 

In our educational structure, decision levels are imbedded in 
educational programs. Decisions related to each program, in turn, 
subsume data elements from combinations of the six classes of 



information across decision levels in order to produce relevant and 



t i rr i e 1 y i n formation. 



To summarize, we have viewed four levels of 
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decision making evolving from the transformation of national goals 
into educational programs. The evaluation concepts outlined enhance 
the necessary program decisions by the production and presentation 
of relevant and timely information gleaned from data elements of the 
six classification categories. 
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